Parallel video processing neural networks

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for parallel processing of video frames using neural networks. One of the methods includes receiving a video sequence comprising a respective video frame at each of a plurality of time steps; and processing the video sequence using a video processing neural network to generate a video processing output for the video sequence, wherein the video processing neural network includes a sequence of network components, wherein the network components comprise a plurality of layer blocks each comprising one or more neural network layers, wherein each component is active for a respective subset of the plurality of time steps, and wherein each layer block is configured to, at each time step at which the layer block is active, receive an input generated at a previous time step and to process the input to generate a block output.

BACKGROUND

This specification relates to processing videos using neural networks.

Neural networks are machine learning models that employ one or morelayers of nonlinear units to predict an output for a received input.Some neural networks include one or more hidden layers in addition to anoutput layer. The output of each hidden layer is used as input to thenext layer in the network, i.e., the next hidden layer or the outputlayer. Each layer of the network generates an output from a receivedinput in accordance with current values of a respective set ofparameters.

Some neural networks are recurrent neural networks. A recurrent neuralnetwork is a neural network that receives an input sequence andgenerates an output sequence from the input sequence. In particular, arecurrent neural network can use some or all of the internal state ofthe network from a previous time step in computing an output at acurrent time step.

SUMMARY

In general, this specification describes techniques for processing videoframes from a video using a video processing neural network that isconfigured such that significant portions of the processing can beparallelized, i.e., performed at the same time rather than needing to beperformed serially. In particular, the video processing neural networkincludes a sequence of components that includes a plurality of layerblocks. At any given time step during the processing of an input videosegment, the operations of the layer blocks can be performed in parallelrather than needing to be performed sequentially, i.e., the operationsof each layer block at any given time step can begin being performedwithout needing to wait for the operations of any other layer block tobe completed for the time step.

Thus in one aspect a method comprises receiving a video sequencecomprising a respective video frame at each of a plurality of timesteps, and processing the video sequence using a video processing neuralnetwork to generate a video processing output for the video sequence. Inimplementations the video processing neural network includes a sequenceof network components. The network components may comprise a pluralityof layer blocks each comprising one or more neural network layers. Insome implementations each component or layer block is active for arespective subset of the plurality of time steps. In implementationseach layer block is configured to, at each time step at which the layerblock is active, receive an input generated at a previous time step andto process the input to generate a block output.

As used herein the video sequence may include or be synonymous with avideo segment as described later. In implementations a network componentmay comprise a layer block or another neural network component. Thus thesequence of network components may comprise a sequence of the layerblocks. The sequence may be such that an output of one component orlayer block is connected to an input of a next component of layer blockin the sequence. The sequence of network components may have an inputnode e.g. to receive the video sequence or a pre-processed version ofthe video sequence, and an output node, e.g. to provide the videoprocessing output. The method may implement parallelized processing inwhich operations are performed by the layer blocks in parallel on datafrom different video frames, e.g. video frames at successive times. Thusprocessing the video sequence may comprise performing the processing oftwo or more of the layer blocks in parallel.

As described later the block output may be a per frame output or may bean output defined by a sequence of multiple video frames (a per segmentoutput). The block output from the parallelized processing may befurther processed to generate a system output. The method, that is theparallelized processing, may be used to perform any video processingtask. For example the neural network layers may be trained to performthe task and the block/system output may provide a task output, forexample to identify or locate objects and/or actions in the videosequence, in which case the task output may comprise data for thispurpose.

As described later, implementations of this method facilitate pipeliningneural network-based video processing and, counter-intuitively, can alsoprovide very low latency processing. This in turn facilitates efficientuse of computational resources and also, for example, low-latencyprocessing of video in real-time. For example some implementationsprocess video frame-by-frame (as opposed to batch processing), andgenerate a causal output, that is an output which at any point in timeis based just on past inputs. Some implementations, as described later,may also have a long temporal receptive field.

As previously described, each component or layer block may be active fora respective subset of the plurality of time steps. The subset of thetime steps may include all the time steps (i.e. the set and subset maycoincide), or the subset may be a so-called proper subset, which doesnot include all the time steps.

In some implementations each layer block is active for a same number orfewer time steps than any layer block before the layer block in thesequence of components. This can reduce the computational requirementsof the method. A layer block may be considered active when it operatesto process data. In many practical applications fast varyingobservations are caused by slow varying factors—for example whenperforming SLAM (Simultaneous Localization and Mapping) scene pose maychange quickly whilst its shape changes slowly or not at all. In anotherexample, when recognizing actions and estimating pose e.g. of a person,pose may change quickly over a period when an action is being performed.Thus later layer blocks in the sequence, which extract higher level,more abstract features, may have a reduced update frequency.

In some implementations, at each time step at which a layer block isactive the layer block does not receive as input any outputs generatedby any other layer blocks at the time step. In other words, theconnections between the layer blocks in the sequence are “diagonal” whenunrolled along a time or frame count dimension. This facilitatespipelining the computations. In some implementations one or more skipconnections may be implemented, which skip one or more layer blocks (orother network components) in the sequence to reduce latency. Thus a skipconnection may be a connection which connects across one or more layerblocks and thus also across one or more layer block processing timesteps. This may therefore involve prediction of an output for a framebased on one or more earlier frames, reducing latency based on therecognition that videos are often predictable over short time intervals.

In some implementations a layer block may comprise an initial layer andone or more additional layers. The initial layer in the block may thenreceive an input generated at the previous time step by a component thatprecedes the layer block in the sequence of components. Optionally eachadditional layer in the block may receive an output generated by one ormore layers at a lower depth level within the same block at the timestep. Here a lower depth level may be a level which is closer to aninput of the sequence of components.

In some implementations of the method a layer block in the sequence ofcomponents may also receive as input a feedback output generated at aprevious time step by one or more components, such as one or more layerblocks, after the layer block in the sequence of components. Thus theremay be feedback of activations from higher layers into lower layerswhich may help, particularly where prediction is involved, by makinghigher level, slower changing representations of the video available tothe lower layers.

One or more of the layer blocks may include a three-dimensionalconvolutional layer with a kernel that has a time dimension of two ormore i.e. that spans two or more time steps or video frames. Each suchlayer block may also receive an input generated at another previous timestep.

The video processing output may be a per-sequence output that includes asingle prediction for the video segment and/or a per-frame output thatincludes a respective prediction for each of multiple frames in thevideo segment.

The video processing neural network may further comprise one or morelayers after the final layer block in the sequence that are configuredto receive the block outputs generated by one or more of the layerblocks and to process the block outputs to generate the video processingoutput.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages.

The neural networks described in this specification can process videosin a manner that is much more computationally efficient than existingneural networks. In particular, the architectures of the describedneural networks allow the computation of the neural network to be highlyparallelized. In particular, at any given time step, the operations ofmultiple layer blocks can be performed in parallel, e.g., on differentcomputing cores, e.g., central processing unit (CPU) cores, or onmultiple different graphics processing units (GPUs). Additionally, thedescribed architectures require minimal computation to generate a newprediction for a newly-received frame. Because of this, the neuralnetworks can generate very low-latency predictions even for tasks thatrequire a prediction to be made for each frame of the video while stillmaintaining a high prediction quality. In particular, the operationsperformed by the neural network are causal, i.e., the prediction for agiven frame does not depend on any frames after the given frame in thevideo, and the operations performed by the layer blocks in the neuralnetwork are parallel and pipelined, i.e., at any given time step,different layer blocks are processing (in parallel) data derived fromdifferent video frames. For example, at one time step, the lowest layerblock in the sequence can process data derived from the video frame atthe time step while, in parallel, another layer block processes dataderived from the video frame at the preceding time step. Because ofthis, the described systems can effectively be used for tasks thatrequire accurate predictions to be made with minimal latency and withoutconsuming excessive amounts of computational resources. In other words,the neural networks described in this specification are speciallyconfigured to be implemented to make online predictions on parallelprocessing hardware.

Additionally, in some cases, because the operations of the neuralnetworks can be performed in parallel and by making use of multi-rateclocks, the network can effectively capture fast-varying features andslow-varying, higher-level features within the video segments.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example video processing system.

FIG. 2 is a diagram that shows example configurations of the neuralnetwork components in the sequence.

FIG. 3 is a diagram that shows a more detailed view of the configurationof the network components when configured for predictivedepth-parallelisation.

FIG. 4 is a diagram that shows additional connectivity options that canbe used to augment a configuration of the neural network components.

FIG. 5 is a flow diagram of an example process for processing a videosegment.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 shows an example video processing system 100. The videoprocessing system 100 is an example of a system implemented as computerprograms on one or more computers in one or more locations, in which thesystems, components, and techniques described below can be implemented.

The video processing system 100 receives an input video segment thatincludes multiple video frames 102, i.e., a respective video frame 102at each of multiple time steps, and uses a video processing neuralnetwork 110 to perform a particular video processing task on thereceived video segment in order to generate a video processing output150.

In some cases, the video processing task is a per-sequence task, i.e., atask that requires a single prediction to be made for the entire videosegment. For example, the video processing task can be an actionclassification task that predicts an action that is being performed overthe course of the frames in the video segment by an agent depicted inthe video segment. In other words, the video processing system 100generates a video processing output 150 that classifies the action beingperformed by the agent in the video segment, i.e., an output thatidentifies one or more actions and, optionally, for each of theidentified actions, a respective probability that the agent ispreforming the identified action in the video segment.

In some other cases, the video processing task is a per-frame task,i.e., a task that requires a separate prediction to be made for eachframe in the video segment or, at a minimum, for a significant portionof the frames in the video segment. For example, the video processingtask can be a pose estimation task that predicts (i.e. estimates), foreach of multiple frames in the video segment, the pose of an agent orobject depicted in the frame, e.g., by generating an output thatidentifies the location of each of multiple joints of the agent orobject in the frame, or an object detection task that predicts(estimates) the location of an object in each frame of multiple framesin the video segment.

More specifically, the video processing neural network 110 is configuredsuch that the video processing system 100 can perform much of theprocessing required for any given video frame in parallel, i.e., insteadof sequentially. This results in the video processing system 100 beingable to maximize available processing power and produce video processingoutputs with minimal latency.

Because of this, the video processing system 100 may be particularlyuseful for being implemented in resource-constrained environments, e.g.,on mobile devices or on other computing devices having limitedprocessing power or memory storage capacity.

Additionally or alternatively, the video processing system 100 may beparticularly useful for performing video processing tasks as part oflarger, latency-sensitive systems, i.e., systems that require videoprocessing tasks to be performed with minimal latency. For example, thevideo processing system 100 may be implemented on-board a self-drivingvehicle or robot and the outputs generated can be used to control theself-driving vehicle or the robot. As another example, the videoprocessing system 100 may be used to analyze and, optionally, modify alive video broadcast. As yet another example, the video processingsystem 100 may be used for automatic features of a camera on a mobiledevice or other user computer, e.g., as part of an auto-focus featurethat determines where to focus a mobile device camera based on the videoprocessing outputs generated by the system 100.

In particular, the video processing neural network 110 includes asequence 104 of neural network components. The sequence 104 includes aplurality of layer blocks 106A-N and an output layer 108. Optionally,the sequence 104 can also include one or more other neural networkcomponents, e.g., one or more conventional neural network layers, e.g.,convolutional layers, max pooling layers, and so on, before the firstlayer block 106A in the sequence 104, after the last layer block 106N inthe sequence and before the output layer, or both. For example, fordense prediction tasks (e.g. predicting a label for each pixel or formultiple regions of the video frame), the sequence 104 can include adecoder neural network head that includes a stack of convolutionallayers, fully-connected layers, or both, that are configured to receivean input and generate the dense prediction map.

Each layer block 106A-N includes one or more neural network layers andis active for a subset of the time steps in the video segment. Inparticular, in some implementations each layer block is active for eachtime step, i.e., the subset for each of the layer blocks is not a propersubset. In some other implementations, each layer block is active foronly a proper subset of the time steps.

More specifically, the neural network 110 may be configured such that,at each time step, all of the operations of the layer blocks that areactive at the time step can be performed in parallel, i.e., none of theactive layer blocks at the time step receive input from any other activelayer blocks at the time step.

In other words, at each time step, the system 100 performs theoperations of some or all of the layer blocks 106A-N in parallel. Thedegree of parallelization can depend on the amount of parallel hardwareresources available to the system 100 at any given time step. Forexample, if the system is implemented on hardware that includes multiplecores such as CPU cores, the system can use different cores to processdifferent ones of the layer blocks 106A-N in parallel at each time step.As another example, if the system is implemented on hardware thatincludes multiple GPUs, the system can use different GPUs to processdifferent layer blocks in parallel at each time step, e.g., byassigning, at each time step, each layer block to one of the multipleGPUs.

Configurations of the neural network 110 are described in more detailbelow with reference to FIGS. 2-4.

FIG. 2 is a diagram 200 that shows example configurations of the neuralnetwork components in the sequence 104 within the video processingneural network 110.

In particular, FIG. 2 shows the processing performed by the videoprocessing neural network at four time steps (0, 1, 2, 3, and 4). Ateach time step, the video processing neural network receives an inputvideo frame I and generates an output (or prediction) y. When the taskis a per-sequence task, the outputs for time steps other than the lasttime step can be discarded or not generated.

In the example of FIG. 2, the sequence includes three layer blocksfollowed by an output layer that generates the output y. While theexample of FIG. 2 shows the first layer block receiving the input videoframe, it will be understood that the input video frame may first beprocessed through one or more other neural network components, e.g., oneor more initial convolutional neural network layers, before beingprovided to the first layer block. Similarly, the output of the lastlayer block may be processed through one or more other neural networkcomponents, e.g., conventionally-configured neural network layers,before being provided to the output layer.

Example configuration 210 (“basic image model”) shows the configurationemployed by conventional systems. In particular, at each time step, thelayer blocks (in the Figure nodes indicated by circles) are connectedvertically within the sequence: the input to each layer block is theoutput of the previous layer block at the same time step, and thenetwork outputs a prediction only after all the layer blocks haveprocessed in sequence the current frame. Thus, the processing of thelayer blocks must be performed serially and cannot be parallelized,i.e., because each layer block requires the preceding layer block in thesequence to generate an output before beginning to process.

Example configuration 220 (“depth-parallelism”) shows a configurationthat allows parallelization and pipelining as described in thisspecification. In the configuration 220, the connections between layerblocks are diagonal instead of vertical: the input to each layer blockat a given time step is the output of the previous layer block from theprevious time step instead of at the given time step. Thus, at any giventime step, each layer block in the network processes its input, passesthe activations to the next layer, and is able to immediately startprocessing the next input available, without waiting for the wholenetwork to finish computation for the current frame.

As can be seen in the configuration 220, at any given time step, theprocessing of the layer blocks is pipeline. In other words, each layerblock processes data for a different video frame, i.e., derived from adifferent video frame, from each other video frame. In other words, atany given time step, each layer block is generating data required tomake the prediction for a different video frame from each other layerblock. The data necessary to make a prediction for a given frame isshown in FIG. 2 as a path of connected nodes starting from the inputvideo frame and ending at the time step at which the output layer makesthe prediction for the frame. For example, at time step 3, the lowestlayer block is processing the video frame I3, i.e., data necessary tomake the prediction y3 for the frame I3, the second lowest layer blockis processing data derived from the video frame I2, i.e., data necessaryto make the prediction y2 for the frame I2, and the third layer block isprocessing data derived from the video frame I1, i.e., data necessary tomake the prediction y1 for the frame I1.

Because there are no dependencies between layer blocks for a given timestep, the system can perform the process for some or all of the layerblocks in parallel at a given time step. As a consequence of thediagonal connectivity, in the configuration 220, the network does notmake the prediction corresponding to a given video frame at the sametime step that the given video frame is provided to the network, thusincurring prediction latency. In particular, in the exampleconfiguration 220, the network is configured to output the predictionfor a video frame at the time step at which the input signal from thevideo frame reaches the output layer, i.e., once the output layerreceives an input generated as a consequence of processing the videoframe. Thus, in the example configuration 220, the prediction latency isthree frames (or, equivalently, three time steps): the network makes theprediction for a given frame three time steps after the given frame isreceived. For example, the network makes the prediction y0 for the frameI0 at time step 3, i.e., the time step at which frame I3 is received.

For some use cases, such a latency is acceptable. However, in otherconfigurations, to reduce the prediction latency, the neural network canbe configured through training to make the prediction for a video framebefore the signal reaches the output layer. In the example configuration230 (“predictive depth-parallelism”), the prediction latency is zero:the network makes the prediction for the given frame at the same timestep at which the frame is received. For example, the network makes theprediction y0 for the frame I0 at time step 0, i.e., the same time stepat which frame I0 is received. By leveraging the fact that videosgenerally are predictable over short horizons and by training the neuralnetwork appropriately, the neural network can still generate accuratepredictions even though the prediction for a given frame is made by theoutput layer based on features of frames preceding the given frame inthe video.

Generally, the neural network can be configured to generate predictionswith a prediction latency of anywhere from 0 to the number of time stepsat which the input signal from the video frame reaches the output layer.In particular, in implementations that require minimal latency, thesystem can sacrifice some amount of prediction accuracy and use a zeroor small prediction latency. In implementations where some latency isacceptable, the system can be configured to have a larger predictionlatency in order to realize some improvement in prediction accuracy.

In implementations where having a low prediction latency is critical,the accuracy of the predictions can be improved by making use of skipconnections. A skip connection passes the output of a given component asinput to another component that is not directly adjacent (in thesequence) to the given component in the neural network. The exampleconfiguration 240 includes a skip connection that passes features fromthe input frame at a given time step to the output layer for use ingenerating the prediction for the video frame at the time step. Forexample, features of the frame I0 are provided to the output layer attime step 0 for use in making the prediction for the prediction y0 forthe frame I0 at time step 0.

Generally, however, skip connections can be inserted along any pathbetween any two components in the sequence that preserves theparallelism and pipelining of the layer block processing. That is, theskip connections can be inserted between any two components in theneural network other than between two layer blocks at the same timestep. For example, a skip connection can be inserted between the outputof one layer block at time step 0 and the output layer (or the otherneural network components preceding the output layer) at time step 1.

FIG. 3 is a diagram 300 that shows a more detailed view of theconfiguration of the network components when configured for predictivedepth-parallelisation.

In particular, FIG. 3 shows two example configurations of the neuralnetwork: a configuration 310 with 3 layer blocks (also referred to as“subnetworks”) and a configuration 320 with 2 layer blocks.

In the configuration 310, each layer block includes two neural networklayers, e.g., convolutional layers. Within the each layer block, theprocessing is sequential, i.e., at each time step the input to thesecond layer in the layer block is the output of the first layer blockin the sequence at the time step. However, as can be seen inconfiguration 310, at any given time step there are no dependenciesbetween the layer blocks. Thus, each layer block can be processed inparallel. For example, at time step 1, layer block 314 receives as inputthe output generated by layer block 312 at time step 0 (and not at timestep 1) and layer block 316 receives as input the output generated bylayer block 314 at time step 0 (and not at time step 1).

Configuration 320 is similar to configuration 310, except that eachlayer block includes three neural network layers and there two totallayer blocks. As can be seen from FIG. 3, there are no dependenciesbetween the layer blocks in configuration 320 at any given time step.

While FIG. 3 shows that each layer block includes a sequence of the samenumber of neural network layers, generally the layers within a givenlayer block can have various configurations and the configuration oflayers can differ between layer blocks.

For example, some or all of the layer blocks can be configured asInception blocks, which include multiple parallel branches of neuralnetwork layers, e.g., branches of convolutional layers with variousfilter sizes and a branch with a max pooling layer, whose outputs areconcatenated to generate the output of the Inception block. Inceptionblocks are described in more detail in Szegedy, C., Liu, W., Jia, Y.,Sermanet, P., Reed, S. E., Anguelov, D., Erhan, D., Vanhoucke, V.,Rabinovich, A.: Going deeper with convolutions. In: IEEE Conference onComputer Vision and Pattern Recognition, CVPR 2015, Boston, Mass., USA,Jun. 7-12, 2015, IEEE Computer Society (2015). In this example, otherlayer blocks may still be sequential (e.g., an initial layer block canbe a sequence of convolutional layers that transforms the input videoframe).

As another example, some or all of the layer blocks can be configured asone or more DenseNet miniblocks, where the miniblocks within each layerblock are densely connected such that every miniblock sends its outputsto all the subsequent miniblocks in the same layer block. DenseNetminiblocks are described in more detail in Huang, G., Liu, Z., van derMaaten, L., Weinberger, K. Q.: Densely connected convolutional networks.In: 2017 IEEE Conference on Computer Vision and Pattern Recognition,CVPR 2017, Honolulu, Hi., USA, Jul. 21-26, 2017, IEEE Computer Society(2017) 2261-2269. A miniblock may comprise a set of neural networkoperations such as one or more of: a 1×1 convolution, an activationfunction, a batch normalization function, and convolutions with one ormore further convolution kernels.

FIG. 4 is a diagram 400 that shows additional connectivity options thatcan be used to augment a configuration of the neural network components.

In particular, FIG. 400 shows three example configurations: atemporalisation configuration 410, a feedback configuration 420, and amulti-rate clock configuration 430. These may be combined.

In the temporalisation configuration 410, some or all of theconvolutional layers in the layer blocks have temporal filters, suchthat they receive additional inputs from past frames. In the exampleconfiguration 410, for example, each layer block receives as input theoutput of the layer block immediately below the layer block both at thecurrent time step and the previous time step, i.e., the filters of thefirst convolutional layer of each block have a temporal size of two.

To apply the temporalisation configuration 410 in the depth-parallel orpredictive depth parallel configurations where there are no verticalconnections between layer blocks, a given layer block can receive asinput the output from the previous block from the past t time steps,where t is the size of the temporal filter of the first layer of a givenblock. In other words, some or all of the convolutional layers in thelayer blocks are implemented as convolutional layers with a causaltemporal filter.

In the feedback configuration 420, the prediction at a given time stepis provided as input to one or more of the layer blocks, e.g., to one ormore of the first layer blocks in the sequence, at the next time step.These additional feedback inputs can be used to provide a simplestarting solution with rich semantics which can be refined by the layerblocks that receive them to improve the next prediction.

In the example configurations shown in FIGS. 2 and 3, each layer blockis active for all of the time steps at which a video frame is received.

In the multi-rate clock configuration 430, on the other hand, not alllayer blocks are active at all of the time steps, i.e., some of thelayer blocks are active for only a proper subset of the time steps.

In particular, features extracted deeper in a neural network tend to bemore abstract and to vary less over time. For example, when tracking anon-rigid moving object, the contours, which are shallow features,change rapidly, but the identity of the object typically does not changeat all. Since not all features change at the same rate as the inputrate, it is then possible to reduce computation by reusing, and notrecomputing, the deeper, more abstract, features. This can beimplemented by having multi-rate clocks as shown in configuration 430:whenever the clock of a layer block does not tick, i.e., the layer blockis not active for a given time step, that layer block does not computeoutputs, instead it reuses the existing ones, i.e., the outputsgenerated by the layer block at the most recent time step at which thelayer block was active. Thus, in the configuration 430, the first layerblock is active for all of the time steps, the second layer block isactive at every other time step, the third layer block is active forevery fourth time step, and the output layer is active at each timestep. In other words, clock rates are reduced by a factor of two betweeneach layer block in the sequence (but the output layer may be alwaysactive). In other multi-rate clock configurations, clock rates canchange differently. Generally, however, each layer block is active foreither the same number or a smaller number of time steps as any earlierlayer block in the sequence.

At each time step at which a given component is active, the component,e.g. layer block, receives as input the output generated by theparticular component, i.e., layer block, immediately before the givencomponent in the sequence at the most recent past time step at which theparticular component was active. Thus, the output layer receives thesame input at time steps 1, 2, 3, and 4 (because the verticalconnections shown in FIG. 4 are not included in depth-parallelconfigurations).

The modifications depicted in configurations 410, 420, and 430 can beapplied to either a depth-parallel or a predictive depth-parallelconfiguration. For example, either the depth-parallel or predictivedepth-parallel configuration can be augmented with one or more feedbackconnections, with temporalisation, or both, to improve predictionaccuracy. As another example, either the depth-parallel or predictivedepth-parallel configuration can be augmented with multi-rate clocks toimprove computational efficiency.

In order for the neural network to generate accurate predictions, thesystem trains the neural network on training data. The training dataincludes a set of training video segments and, for each video segment, acorresponding ground truth output. When the task requires a singleprediction, the ground truth output includes a single output for eachvideo segment. When the task requires predictions for each of multiplesframes, the ground truth output includes a respective output for each ofthe frames for which an output is required; the training may thus beframe-by-frame.

The system trains the neural network, i.e., determines trained values ofthe parameters of the neural network, on the training data to optimizean objective function, e.g., to minimize a loss function, that measuresthe error between the outputs generated by the neural network and theground truth outputs. For example, the error can be a cross-entropyloss. In order to configure the neural network for predictive depthparallelism, the objective function measures the error between theground truth output for a given frame and the prediction made at thetime step that is l time steps after the given frame in the trainingsegment, where l is the prediction latency.

The system can optimize this objective function using conventionalsupervised learning techniques, e.g., stochastic gradient descent withmomentum or Adam.

While the proposed parallel configurations reduce latency andcomputational resource consumption, their computational depth for thecurrent frame at the time at which they produce an output for thecurrent frame is also reduced compared to their fully sequentialcounterparts. Additionally, some configurations are designed to re-usefeatures from previous states through the multi-rate clocks mechanism.

In some cases, the system accounts for these factors by augmenting thetraining of the neural network by making use of a conventionalsequential neural network that has been trained to perform the same taskand that has the same overall architecture as the parallel neuralnetwork. Because the conventional sequential neural network does notemploy the parallelization techniques described in this specification,all of its layers always have access to fresh features extracted fromthe current frame.

During the training of the parallel neural network, the objectivefunction can include an additional term that encourage activations tomatch those of the sequential model for some predetermined layers (whilestill optimizing the original objective function). Therefore, theparallel model can be encouraged generate abstract features that matchhow the features would have looked, had the information from the currentframe been available. This additional term can be implemented as, forexample, for each predetermined layer, a sum of the normalized Euclideandistance or other distance measure between the corresponding activationsof the sequential and parallel models.

FIG. 5 is a flow diagram of an example process 500 for processing avideo segment. For convenience, the process 500 will be described asbeing performed by a system of one or more computers located in one ormore locations. For example, a video processing system, e.g., the videoprocessing system 100 of FIG. 1, appropriately programmed, can performthe process 500.

The system can perform the process 500 for each time step in a videosegment to generate a video processing output for the video segment.

The system receives an input video frame at the time step (step 502).

The system identifies which components in the sequence are active at thetime step (step 504). In particular, when multi-rate clocks are notbeing used, each component is active at each time step. When multi-rateclocks are being used, on the other hand, some components are active foronly a proper subset of the time steps.

The system performs the processing of each active component for the timestep (step 506) in accordance with the configuration of the neuralnetwork. In particular, each active layer block (other than the firstlayer block in the sequence) receives as input at the time step anoutput generated by a preceding layer block at one or more earlier timesteps at which the preceding layer block was active, i.e., and does notreceive any outputs generated by any other layer blocks at the timestep. Because there are no dependencies between the operations performedby the active layer blocks at any given time step, when multiple layerblocks are active at the time step, the system performs the operationsof two or more of the active layer blocks in parallel, e.g., by usingmultiple CPU cores to perform the operations or by using multiple GPUsto perform the operations. Additionally, the operations performed by thesystem are pipelined, i.e., each layer block that is active at the timestep is operating on and generating data necessary to make theprediction for a different video frame.

While this specification describes processing videos, the describedarchitectures and training schemes can be employed for neural networksthat process other types of sequential data that have a relatively largeamount of data per-timestep and for which it is beneficial to applyneural networks having multiple layer blocks. For example, the describedtechniques can be used to process and make predictions from streams ofLIDAR (light detection and ranging) data or other sensor data, e.g.,data collected by sensors of a robot or an autonomous vehicle.

This specification uses the term “configured” in connection with systemsand computer program components. For a system of one or more computersto be configured to perform particular operations or actions means thatthe system has installed on it software, firmware, hardware, or acombination of them that in operation cause the system to perform theoperations or actions. For one or more computer programs to beconfigured to perform particular operations or actions means that theone or more programs include instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the operations oractions.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non transitory storage medium for execution by, or to controlthe operation of, data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them. Alternatively or in addition, the programinstructions can be encoded on an artificially generated propagatedsignal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. For example the system of FIG. 1 may bealternatively implemented partially or wholly in dedicated hardware.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can alsobe, or further include, special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can optionally include, in additionto hardware, code that creates an execution environment for computerprograms, e.g., code that constitutes processor firmware, a protocolstack, a database management system, an operating system, or acombination of one or more of them.

A computer program, which may also be referred to or described as aprogram, software, a software application, an app, a module, a softwaremodule, a script, or code, can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages; and it can be deployed in any form, including as astand alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A program may, but neednot, correspond to a file in a file system. A program can be stored in aportion of a file that holds other programs or data, e.g., one or morescripts stored in a markup language document, in a single file dedicatedto the program in question, or in multiple coordinated files, e.g.,files that store one or more modules, sub programs, or portions of code.A computer program can be deployed to be executed on one computer or onmultiple computers that are located at one site or distributed acrossmultiple sites and interconnected by a data communication network.

In this specification, the term “database” is used broadly to refer toany collection of data: the data does not need to be structured in anyparticular way, or structured at all, and it can be stored on storagedevices in one or more locations. Thus, for example, the index databasecan include multiple collections of data, each of which may be organizedand accessed differently.

Similarly, in this specification the term “engine” is used broadly torefer to a software-based system, subsystem, or process that isprogrammed to perform one or more specific functions. Generally, anengine will be implemented as one or more software modules orcomponents, installed on one or more computers in one or more locations.In some cases, one or more computers will be dedicated to a particularengine; in other cases, multiple engines can be installed and running onthe same computer or computers.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby special purpose logic circuitry, e.g., an FPGA or an ASIC, or by acombination of special purpose logic circuitry and one or moreprogrammed computers.

Computers suitable for the execution of a computer program can be basedon general or special purpose microprocessors or both, or any other kindof central processing unit. Generally, a central processing unit willreceive instructions and data from a read only memory or a random accessmemory or both. The essential elements of a computer are a centralprocessing unit for performing or executing instructions and one or morememory devices for storing instructions and data. The central processingunit and the memory can be supplemented by, or incorporated in, specialpurpose logic circuitry. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device, e.g., a universalserial bus (USB) flash drive, to name just a few.

Computer readable media suitable for storing computer programinstructions and data include all forms of non volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto optical disks; andCD ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's device in response to requests received from the web browser.Also, a computer can interact with a user by sending text messages orother forms of message to a personal device, e.g., a smartphone that isrunning a messaging application, and receiving responsive messages fromthe user in return.

Data processing apparatus for implementing machine learning models canalso include, for example, special-purpose hardware accelerator unitsfor processing common and compute-intensive parts of machine learningtraining or production, i.e., inference, workloads.

Machine learning models can be implemented and deployed using a machinelearning framework, e.g., a TensorFlow framework, a Microsoft CognitiveToolkit framework, an Apache Singa framework, or an Apache MXNetframework.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface, a web browser, or anapp through which a user can interact with an implementation of thesubject matter described in this specification, or any combination ofone or more such back end, middleware, or front end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the device, which acts as a client.Data generated at the user device, e.g., a result of the userinteraction, can be received at the server from the device.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or on the scope of what may be claimed, but rather asdescriptions of features that may be specific to particular embodimentsof particular inventions. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially be claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and recited inthe claims in a particular order, this should not be understood asrequiring that such operations be performed in the particular ordershown or in sequential order, or that all illustrated operations beperformed, to achieve desirable results. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system modules and components in the embodimentsdescribed above should not be understood as requiring such separation inall embodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

1. A method comprising: receiving a video sequence comprising arespective video frame at each of a plurality of time steps; andprocessing the video sequence using a video processing neural network togenerate a video processing output for the video sequence, wherein thevideo processing neural network includes a sequence of networkcomponents, wherein the network components comprise a plurality of layerblocks each comprising one or more neural network layers, wherein eachcomponent is active for a respective subset of the plurality of timesteps, and wherein each layer block is configured to, at each time stepat which the layer block is active, receive an input generated at aprevious time step and to process the input to generate a block output.2. The method of claim 1, wherein each layer block is active for a samenumber or fewer time steps than any layer block before the layer blockin the sequence of components.
 3. The method of claim 1, wherein one ormore of the layer blocks comprise an initial layer and one or moreadditional layers, wherein the initial layer in each layer blockreceives an input generated at the previous time step by a componentthat precedes the layer block in the sequence of components, and whereineach additional layer in each layer block receives an output generatedby one or more layers at a lower depth level within the same layer blockat the time step.
 4. The method of claim 1, wherein one or more of thelayer blocks in the sequence of components also receive as input afeedback output generated at a previous time step by one or morecomponents after the block in the sequence of components.
 5. The methodof claim 1, wherein one or more of the layer blocks includethree-dimensional convolutional layers with kernels that have a timedimension of two or more.
 6. The method of claim 5, wherein each blockthat includes three-dimensional convolutional layers with kernels thathave a time dimension of two or more also receives an input generated atanother previous time step.
 7. The method of claim 1, wherein the videoprocessing output is a per-sequence output that includes a singleprediction for the video segment.
 8. The method of claim 1, wherein thevideo processing output is a per-frame output that includes a respectiveprediction for each of multiple frames in the video segment.
 9. Themethod of claim 1, wherein the video processing neural network furthercomprises one or more layers after the final layer block in the sequencethat are configured to receive the block outputs generated by one ormore of the layer blocks and to process the block outputs to generatethe video processing output.
 10. The method of claim 1, wherein theprocessing comprises, at each time step: performing the processing oftwo or more of the layer blocks in parallel.
 11. The method of claim 1,wherein at each time step at which the layer block is active, the layerblock does not receive as input any outputs generated by any other layerblocks at the time step.
 12. The method of claim 1, wherein at each timestep, each layer that is active operates on data derived from adifferent video frame than each other active layer block.
 13. A systemcomprising one or more computers and one or more storage devices storinginstructions that when executed by the one or more computers cause theone or more computers to perform operations comprising: receiving avideo sequence comprising a respective video frame at each of aplurality of time steps; and processing the video sequence using a videoprocessing neural network to generate a video processing output for thevideo sequence, wherein the video processing neural network includes asequence of network components, wherein the network components comprisea plurality of layer blocks each comprising one or more neural networklayers, wherein each component is active for a respective subset of theplurality of time steps, and wherein each layer block is configured to,at each time step at which the layer block is active, receive an inputgenerated at a previous time step and to process the input to generate ablock output.
 14. One or more computer storage media storinginstructions that when executed by one or more computers cause the oneor more computers to perform operations comprising: receiving a videosequence comprising a respective video frame at each of a plurality oftime steps; and processing the video sequence using a video processingneural network to generate a video processing output for the videosequence, wherein the video processing neural network includes asequence of network components, wherein the network components comprisea plurality of layer blocks each comprising one or more neural networklayers, wherein each component is active for a respective subset of theplurality of time steps, and wherein each layer block is configured to,at each time step at which the layer block is active, receive an inputgenerated at a previous time step and to process the input to generate ablock output.
 15. The system of claim 13, wherein each layer block isactive for a same number or fewer time steps than any layer block beforethe layer block in the sequence of components.
 16. The system of claim13, wherein one or more of the layer blocks comprise an initial layerand one or more additional layers, wherein the initial layer in eachlayer block receives an input generated at the previous time step by acomponent that precedes the layer block in the sequence of components,and wherein each additional layer in each layer block receives an outputgenerated by one or more layers at a lower depth level within the samelayer block at the time step.
 17. The system of claim 13, wherein one ormore of the layer blocks in the sequence of components also receive asinput a feedback output generated at a previous time step by one or morecomponents after the block in the sequence of components.
 18. The systemof claim 13, wherein the processing comprises, at each time step:performing the processing of two or more of the layer blocks inparallel.
 19. The system of claim 13, wherein at each time step at whichthe layer block is active, the layer block does not receive as input anyoutputs generated by any other layer blocks at the time step.
 20. Thesystem of claim 13, wherein at each time step, each layer that is activeoperates on data derived from a different video frame than each otheractive layer block.